This document includes all information referenced in the main paper. We show full details of the process of analysis and the steps taken to ensure the statistical viability of those reported, documenting what the pre-registered models were and providing justification in cases where the final models differed.

As a supplementary analysis, we also show how differences between the adverts impacted key outcome variables. However, keep in mind the mixed-effect models used tested if the main effects were generalisable beyond specific stimuli-level differences. The advert-level results are provided here for context but do not affect the generalisability or statistical robustness of our primary conclusions.

Model Fit Considerations

For hypotheses 1-3 we used mixed-effect modelling techniques. It is essential for random effect model structures to be theoretically motivated and justified (Brown, 2021). During the pre-registration stage, it was reasoned the inclusions of two separate random intercept terms set for participants (1,322 grouping levels) and each material (four grouping levels) were the most suitable due to how each participant would approach the materials with a different evaluative baseline, and in turn each piece of material differed in its message and format creating its own baseline through which it would be judged from. However, during the process of data analysis, some key considerations came to light which concerned best practice for multi-level models, a topic debated across the literature.

For example, it is recognised that not accounting for correlations within the random effects increases the risk of a type 1 error when estimated fixed effects (Matuschek et al., 2017). For this reason, Barr et al. (2013) recommend always opting for the ‘maximal’ model (with the highest complexity) that can be theoretically justified. However, increasing model complexity also reduces statistical power, requiring consideration as to whether it can be deemed a necessary trade off (Matuschek et al., 2017). Matuschek et al. (2017) recommend using estimators of model fit, such as the Likelihood Ratio Test (LRT) and the Akaike Information Criterion (AIC) to check if increasing complexity substantially improves model fit.

This point is relevant because, during the analysis, other theoretically viable model structures were identified that supported a more complex random effect structure. For example, adding a random slope term would account for whether the effect of a digital imprint varied depending on the unique characteristics of the material—i.e., if the digital imprint was effective on some materials but not others. Consequently, various theoretically justified models were tested to find a random effects structure that balanced complexity with good model fit. Following the recommendations in Matuschek et al. (2017), to assess model suitability we used likelihood ratio tests (LRT) to compare nested models (e.g., the addition of interaction terms between fixed effects) and AIC to compare models with different random effect structures. Models with lower AIC values indicate a better relative fit for the data (Cavanaugh and Neath, 2019), helping determine if additional complexity improves (or weakens) model performance (Matuschek et al., 2017). Out of all model variants tested, the model with the lowest AIC - if it would converge - was used as a baseline. Those with an AIC within two units of the lowest were considered viable, while those between two and ten units of the lowest were deemed less supported. Models with AIC values ten units or more above the best model were considered unsupported as a model structure (for an explanation see Cavanaugh and Neath, 2019). Full details of alternative models are provided, and deviations from pre-registered models are transparently discussed.

Criteria for determining model support (Cavanaugh & Neath, 2019):

  • Model with the lowest AIC = best model fit
  • Within 0-2 of lowest model = viable model and should still be considered if it better reflects the theoretical motivations of the analysis
  • Within 2-10 of the lowest model = low support for the model
  • 10+ of the lowest model = no support for this model

Preparing the dataset

Libraries used

library(dplyr)
## 
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union
library(tidyr)
library(stringr)
library(purrr)

Reading in data and exclusions

data <- read.csv("rawdata_with_wranglecode/main_data.csv")

#change name of row number identifier

data <- data %>% 
  mutate(id = row_number())

#removing missing data rows from dataset (12 in total) for participants who did not complete the survey

rows_with_blank_in_name <- grepl("^\\s*$", data$EPE_5)
df_with_blanks_in_name <- data[rows_with_blank_in_name, ]

data <- data[!rows_with_blank_in_name, ]

#removing row numbers from the dataset where both attention checks were failed (the process that identified these row numbers can be viewed in 'data_quality_check.Rmd')

data <- data %>%
  filter(!(row_number() %in% c(420, 948, 1288, 1315)))

#removing exclusions based on half the median time, 360 seconds

data <- data %>%
  filter(Duration..in.seconds. >= 360)

#this should leave a data-frame with 1322 observations
#extract relevant variables into new data frame

data <- data %>%
  select(id, Training.condition, Advert.1, Advert.2, Advert.3, Advert.4, starts_with("PK"), starts_with("agree"), starts_with("informed"), starts_with("accurate"), starts_with("believable"), starts_with("trustworthy"), starts_with("factual"), election_reg, recall_num, recall_name, starts_with("useful"), reg_know, starts_with("EPE"), starts_with("general_confidence"), starts_with("institution"), democracy, political_interest, external_efficacy, internal_efficacy, starts_with("SM"), partyID, age_sample, gender, education, Ethnicity.simplified)

Agree/disagree item transformations

The code below will convert all variables with response measurement of strongly disagree to strongly agree from a character variable to a numerical scale of 1-7. One item also needs to be reverse scored:

  • Informed item 3: ‘I am not sure who is behind this material’

There are also attention checks in the dataset that need to be removed once exclusions have been dealt with:

  • informed_2_5
  • informed_2imprint_5
  • EPE_5

Below creates a functions that will be applied to all agree-disagree response formats in the dataset - all ones that start with PK, agree, informed, EPE and general_confidence. The second function then reverse scores informed item three across the eight advert variations.

#converting to numeric variables from character for agree - disagree
#Persuasion knowledge measures accidentally has a slightly different response option compared to other measures, meaning 2 conversion functions are needed. Instead of 'somewhat' they got 'slightly'.

convert_numeric1 <- function(response) {
  
  # Trim leading and trailing whitespace and convert to lowercase
  response_cleaned <- tolower(trimws(response))
  
  # Define the mapping with all lowercase keys
  mapping <- c(
    "strongly disagree" = 1,
    "disagree" = 2,
    "slightly disagree" = 3,
    "neither agree nor disagree" = 4,
    "slightly agree" = 5,
    "agree" = 6,
    "strongly agree" = 7
  )
  
  # Return the mapped value, or NA if the response does not match
  return(ifelse(!is.na(mapping[response_cleaned]), mapping[response_cleaned], NA))
}

convert_numeric2 <- function(response) {
  
  # Trim leading and trailing whitespace and convert to lowercase
  response_cleaned <- tolower(trimws(response))
  
  # Define the mapping with all lowercase keys
  mapping <- c(
    "strongly disagree" = 1,
    "disagree" = 2,
    "somewhat disagree" = 3,
    "neither agree nor disagree" = 4,
    "somewhat agree" = 5,
    "agree" = 6,
    "strongly agree" = 7
  )
  
  # Return the mapped value, or NA if the response does not match
  return(ifelse(!is.na(mapping[response_cleaned]), mapping[response_cleaned], NA))
}

#applying this function to the data frame (two separate functions to account for differences in response options)
         
data <- data %>%
  mutate(across(starts_with("PK"), convert_numeric1))

data <- data %>%
  mutate(across(c(starts_with("informed"), starts_with("agree"), starts_with("EPE"), starts_with("general")), ~convert_numeric2(.x)))

#reverse scoring informed item 3

reverse_code <- function(response) {
  # Define the mapping from original to reversed scores
  mapping <- c(1, 2, 3, 4, 5, 6, 7)
  names(mapping) <- c(7, 6, 5, 4, 3, 2, 1)
  
  # Use the response as a name to look up in the mapping
  return(as.numeric(names(mapping)[match(response, mapping)]))
}

data <- data %>%
  mutate(across(c(informed_1_3, informed_1imprint_3, informed_2_3, informed_2imprint_3, informed_3_3, informed_3imprint_3, informed_4_3, informed_4imprint_3), ~reverse_code(.x)))

#removing the attention check columns from the dataset

data <- data %>%
  select(-informed_2_5, -informed_2imprint_5, -EPE_5)

Variable tranformations for both RM and IM dataframes

The code below conducts the following transformations to the variables that will be present in both the repeated measures and independent measures data frames so they are ready to be analysed:

  • Transformed to a factor: advert.1, advert.2, advert.3, advert.4, Training.condition, reg_know, SM_use, starts with: SM_frequency, party_ID, gender, education

  • Transformed to a numerical variable: election_reg, starts with: useful_rank, starts with: institution_trust, democracy, political_interest, external_efficacy, internal_efficacy, age

Some variables will only be present in the repeated measures data frame and will be created later.

#creating factor variables through use of a function

convert_to_factor <- function(df, cols) {
  df %>%
    mutate(across(all_of(cols), as.factor))
}

data <- data %>%
  convert_to_factor(c("Advert.1", "Advert.2", "Advert.3", "Advert.4", "SM_frequency_1", "SM_use", "Training.condition", "reg_know", "SM_use", "partyID", "gender", "education", "Ethnicity.simplified"))

#Setting reference groups for: reg_know, SM_use, SM_frequency, gender, education, ethnicity

#regulation knowledge

reg_response_order <- c("There are no regulatory controls on any type of political advertising during UK elections", "All political advertising is regulated by rules set by the UK government, but there is one set of rules for advertising on television and radio and a different set of rules for advertising on the internet and social media", "All political advertising (whether on television, radio, in newspapers or the internet) is subject to the same rules set by the UK government", "Not sure")

data <- data %>%
  mutate(across(reg_know, ~factor(.x, levels = reg_response_order)))

#Social media use
  
use_response_order <- c("None, No time at all ", "Less than 1/2 hour ", "1/2 hour to 1 hour ", "1 to 2 hours ",  "Not sure")

data <- data %>%
  mutate(across(SM_use, ~factor(.x, levels = use_response_order)))
  
#SM frequency use
  
freq_response_order <- c("Never",
                         "Less than once a week",
                         "Once a week\t",
                         "Once every couple of days\t",
                         "Once a day\t",
                         "2-5 times a day",
                         "More than five times a day\t")

data <- data %>%
  mutate(across(SM_frequency_1, ~factor(.x, levels = freq_response_order)))

#gender, female as reference

gender_response_order <- c("Female", "Male", "Non-binary / third gender", "Prefer not to say")

data <- data %>%
  mutate(across(gender, ~factor(.x, levels = gender_response_order)))

#Education level, postgrad as reference

ed_response_order <- c("Postgraduate (e.g. M.Sc, Ph.D)", "Undergraduate University (e.g. BA, B.Sc, B.Ed)", "A-level, or equivalent", "GCSE level, or equivalent", "Other, please specify", "No formal qualifications")

data <- data %>%
  mutate(across(education, ~factor(.x, levels = ed_response_order)))

#Ethnicity level, white as reference

ethn_response_order <- c("White", "Asian", "Black", "Mixed", "Other")

data <- data %>%
  mutate(across(Ethnicity.simplified, ~factor(.x, levels = ethn_response_order)))
#Need to first change response options from categories to numbers for: election_reg, institution_trust, democracy, political_interest, internal_efficacy, external_efficacy, age

#Confidence in electoral regulation

data <- data %>%
  mutate(election_reg = case_when(
    election_reg == "Completely insufficient" ~ 1,
    election_reg == "Mostly insufficient" ~ 2,
    election_reg == "Slightly insufficient" ~ 3,
    election_reg == "No opinion/not sure" ~ 4,
    election_reg == "Slightly sufficient" ~ 5,
    election_reg == "Mostly sufficient" ~ 6,
    election_reg == "Completely sufficient" ~ 7
  ))

#Converting 'democracy' to a numeric variable

data <- data %>%
  mutate(democracy = case_when(
    democracy == "Very dissatisfied" ~ 1,
    democracy == "A little dissatisfied" ~ 2,
    democracy == "Fairly satisfied" ~ 3,
    democracy == "Very satisfied" ~ 4
  ))

#converting political interest to a numerical variable

data <- data %>%
  mutate(political_interest = case_when(
    political_interest == "Not at all interested" ~ 1,
    political_interest == "Not very interested" ~ 2,
    political_interest == "Slightly interested" ~ 3,
    political_interest == "Fairly interested" ~ 4,
    political_interest == "Very interested " ~ 5
  ))

#converting internal and external efficacy to numeric, 5 options

data <- data %>%
  mutate(internal_efficacy = case_when(
    internal_efficacy == "Not at all " ~ 1,
    internal_efficacy == "A little " ~ 2,
    internal_efficacy == "A moderate amount  " ~ 3,
    internal_efficacy == "A lot " ~ 4,
    internal_efficacy == "A great deal " ~ 5
  ))

data <- data %>%
  mutate(external_efficacy = case_when(
    external_efficacy == "Not at all " ~ 1,
    external_efficacy == "A little " ~ 2,
    external_efficacy == "A moderate amount  " ~ 3,
    external_efficacy == "A lot " ~ 4,
    external_efficacy == "A great deal " ~ 5
  ))

#creating numeric variables through the use of a function

convert_to_numeric <- function(df, cols) {
  df %>%
    mutate(across(all_of(cols), as.numeric))
}

#age

data$age_sample <- as.numeric(data$age_sample)

#Convert all other variables to numeric

data <- data %>%
  convert_to_numeric(c("useful_rank_1", "useful_rank_2", "useful_rank_3", "useful_rank_4", "useful_rank_5", "useful_rank_6"))

Recall variable transformations

Transformation of recall variables:

  • Recall_num: two new columns need to be created specifying those who picked ‘not sure’ versus those who chose an answer, then those who were correct, chose 2, and those who were incorrect.

  • Recall_name: 8 potential columns will need to be created with a binary response, indicating whether each name option was identified e.g. ‘common sense collective’.

  • The correct identification options are:

    • Common sense collective - advert 1
    • Breaking barriers alliance - advert 2
    • Speak freely Inc.- advert 3
    • Campaign for a better Britain - advert 4

  • Incorrect options

    • Future first
    • The peoples movement
    • Voice for the people
    • Hope something - removed from qualtrics and replaced with ad 4
    • All together
#Recall number transformation for correct/incorrect response

data <- data %>%
  mutate(recall_correct = 
           case_when(
             recall_num == 2 ~ "correct",
             TRUE ~ "incorrect"
           ))

#Recall name transformation, correct responses

data <- data %>%
  mutate(CSC = case_when(
    str_detect(recall_name, "Common Sense Collective") ~ 1,
    TRUE ~ 0
  ))

data <- data %>%
  mutate(BBA = case_when(
    str_detect(recall_name, "Breaking Barriers Alliance") ~ 1,
    TRUE ~ 0
  ))

data <- data %>%
  mutate(SFI = case_when(
    str_detect(recall_name, "Speak Freely Inc") ~ 1,
    TRUE ~ 0
  ))

data <- data %>%
  mutate(CBB = case_when(
    str_detect(recall_name, "Campaign for a better Britain") ~ 1,
    TRUE ~ 0
  ))

#incorrect responses

data <- data %>%
  mutate(FF = case_when(
    str_detect(recall_name, "Future First") ~ 1,
    TRUE ~ 0
  ))

data <- data %>%
  mutate(TPM = case_when(
    str_detect(recall_name, "The People’s movement") ~ 1,
    TRUE ~ 0
  ))

data <- data %>%
  mutate(VFP = case_when(
    str_detect(recall_name, "Voice for the People") ~ 1,
    TRUE ~ 0
  ))

data <- data %>%
  mutate(AT = case_when(
    str_detect(recall_name, "All Together") ~ 1,
    TRUE ~ 0
  ))

#number of correct names recalled, name_correct

data <- data %>%
  mutate(name_correct = CSC + BBA + SFI + CBB)

#number of incorrect names recalled, name_incorrect

#add incorrect columns together

data <- data %>%
  mutate(name_incorrect = FF + TPM + VFP + AT)

#convert campaign names to factors

data <- data %>%
  convert_to_factor(c("recall_correct", "CSC", "BBA", "SFI", "CBB", "FF", "TPM", "VFP", "AT"))

Repeated measures dataframe

The code below turns the wide data into long data, creating 4 rows for each participant and only one column for each of the outcome variables: persuasion knowledge, political goal, informedness, agreement, believability, trustworthiness, accurateness, factual. Extra columns also specify the advert viewed and the version (imprint or no imprint).

#create a new dataframe with only the repeated measures (post-advert) variables

RM <- data %>%
  select(id, starts_with("Advert."), starts_with("PK"), starts_with("agree"), starts_with("informed"), starts_with("accurate"), starts_with("believable"), starts_with("trustworthy"), starts_with("factual"))

#when first converted into long data, eight rows are generated for each participant for the eight different advert variations, but many columns contain NA.

#persuasion knowledge df, each item separate

PK1_long <- RM %>%
  select(id, starts_with("Advert."), PK_1_1, PK_1imprint_1, PK_2_1, PK_2imprint_1, PK_3_1, PK_3imprint_1, PK_4_1, PK_4imprint_1) %>%
  pivot_longer(
    cols = c(PK_1_1, PK_1imprint_1, PK_2_1, PK_2imprint_1, PK_3_1, PK_3imprint_1, PK_4_1, PK_4imprint_1),
    names_to = "PK1",
    values_to = "PK1_value"
  )

PK2_long <- RM %>%
  select(id, starts_with("Advert."), PK_1_2, PK_1imprint_2, PK_2_2, PK_2imprint_2, PK_3_2, PK_3imprint_2, PK_4_2, PK_4imprint_2) %>%
  pivot_longer(
    cols = c(PK_1_2, PK_1imprint_2, PK_2_2, PK_2imprint_2, PK_3_2, PK_3imprint_2, PK_4_2, PK_4imprint_2),
    names_to = "PK2",
    values_to = "PK2_value"
  )

PK3_long <- RM %>%
  select(id, starts_with("Advert."), PK_1_3, PK_1imprint_3, PK_2_3, PK_2imprint_3, PK_3_3, PK_3imprint_3, PK_4_3, PK_4imprint_3) %>%
  pivot_longer(
    cols = c(PK_1_3, PK_1imprint_3, PK_2_3, PK_2imprint_3, PK_3_3, PK_3imprint_3, PK_4_3, PK_4imprint_3),
    names_to = "PK3",
    values_to = "PK3_value"
  )

PK4_long <- RM %>%
  select(id, starts_with("Advert."), PK_1_4, PK_1imprint_4, PK_2_4, PK_2imprint_4, PK_3_4, PK_3imprint_4, PK_4_4, PK_4imprint_4) %>%
  pivot_longer(
    cols = c(PK_1_4, PK_1imprint_4, PK_2_4, PK_2imprint_4, PK_3_4, PK_3imprint_4, PK_4_4, PK_4imprint_4),
    names_to = "PK4",
    values_to = "PK4_value"
  )


#political goal df, informed item 1

PG_long <- RM %>%
  select(id, starts_with("Advert."), informed_1_1, informed_1imprint_1, informed_2_1, informed_2imprint_1, informed_3_1, informed_3imprint_1, informed_4_1, informed_4imprint_1) %>%
  pivot_longer(
    cols = c(informed_1_1, informed_1imprint_1, informed_2_1, informed_2imprint_1, informed_3_1, informed_3imprint_1, informed_4_1, informed_4imprint_1),
    names_to = "political_goal",
    values_to = "PG_value"
  )

#informed df, each item separate

informed2_long <- RM %>%
  select(id, starts_with("Advert."), informed_1_2, informed_1imprint_2, informed_2_2, informed_2imprint_2, informed_3_2, informed_3imprint_2, informed_4_2, informed_4imprint_2) %>%
  pivot_longer(
    cols = c(informed_1_2, informed_1imprint_2, informed_2_2, informed_2imprint_2, informed_3_2, informed_3imprint_2, informed_4_2, informed_4imprint_2),
    names_to = "informed2",
    values_to = "informed2_value"
  )

informed3_long <- RM %>%
  select(id, starts_with("Advert."), informed_1_3, informed_1imprint_3, informed_2_3, informed_2imprint_3, informed_3_3, informed_3imprint_3, informed_4_3, informed_4imprint_3) %>%
  pivot_longer(
    cols = c(informed_1_3, informed_1imprint_3, informed_2_3, informed_2imprint_3, informed_3_3, informed_3imprint_3, informed_4_3, informed_4imprint_3),
    names_to = "informed3",
    values_to = "informed3_value"
  )

informed4_long <- RM %>%
  select(id, starts_with("Advert."), informed_1_4, informed_1imprint_4, informed_2_4, informed_2imprint_4, informed_3_4, informed_3imprint_4, informed_4_4, informed_4imprint_4) %>%
  pivot_longer(
    cols = c(informed_1_4, informed_1imprint_4, informed_2_4, informed_2imprint_4, informed_3_4, informed_3imprint_4, informed_4_4, informed_4imprint_4),
    names_to = "informed4",
    values_to = "informed4_value"
  )

#agreement df

agree_long <- RM %>%
  select(id, starts_with("Advert."), starts_with("agree")) %>%
  pivot_longer(
    cols = starts_with("agree"),
    names_to = "agree",
    values_to = "agree_value"
  )

#trustworthy df

trustworthy_long <- RM %>%
  select(id, starts_with("Advert."), starts_with("trustworthy")) %>%
  pivot_longer(
    cols = starts_with("trustworthy"),
    names_to = "trustworthy",
    values_to = "trustworthy_value"
  )

#believability df

believe_long <- RM %>%
  select(id, starts_with("Advert."), starts_with("believable")) %>%
  pivot_longer(
    cols = starts_with("believable"),
    names_to = "believable",
    values_to = "believable_value"
  )

#accurateness df

accurate_long <- RM %>%
  select(id, starts_with("Advert."), starts_with("accurate")) %>%
  pivot_longer(
    cols = starts_with("accurate"),
    names_to = "accurate",
    values_to = "accurate_value"
  )

#factual df

factual_long <- RM %>%
  select(id, starts_with("Advert."), starts_with("factual")) %>%
  pivot_longer(
    cols = starts_with("factual"),
    names_to = "factual",
    values_to = "factual_value"
  )

#Create two new variables in each indicating advert type and version viewed, so that the dataframes can be merged by these two columns

#Below is three functions that can be applied to each df to create new variables.

# Function to add 'advert' and 'version' based on patterns in a specified column
add_advert_version <- function(data, column_name) {
  data %>%
    mutate(
      advert = case_when(
        str_detect(!!sym(column_name), "1") ~ "advert.1",
        str_detect(!!sym(column_name), "2") ~ "advert.2",
        str_detect(!!sym(column_name), "3") ~ "advert.3",
        str_detect(!!sym(column_name), "4") ~ "advert.4",
        TRUE ~ NA_character_
      ),
      version = case_when(
        str_detect(!!sym(column_name), "imprint") ~ 1,
        TRUE ~ 0
      )
    ) 
}

#apply function for agree, trust, believe, factual, accurate

agree_long <- add_advert_version(agree_long, "agree")
trustworthy_long <- add_advert_version(trustworthy_long, "trustworthy")
believe_long <- add_advert_version(believe_long, "believable")
accurate_long <- add_advert_version(accurate_long, "accurate")
factual_long <- add_advert_version(factual_long, "factual")

#PK function

PK_advert_version <- function(data, column_name) {
  data %>%
    mutate(
      advert = case_when(
        str_detect(!!sym(column_name), "PK_1") ~ "advert.1",
        str_detect(!!sym(column_name), "PK_2") ~ "advert.2",
        str_detect(!!sym(column_name), "PK_3") ~ "advert.3",
        str_detect(!!sym(column_name), "PK_4") ~ "advert.4",
        TRUE ~ NA_character_
      ),
      version = case_when(
        str_detect(!!sym(column_name), "imprint") ~ 1,
        TRUE ~ 0
      )
    ) 
}

PK1_long <- PK_advert_version(PK1_long, "PK1")
PK2_long <- PK_advert_version(PK2_long, "PK2")
PK3_long <- PK_advert_version(PK3_long, "PK3")
PK4_long <- PK_advert_version(PK4_long, "PK4")

#informed function

in_advert_version <- function(data, column_name) {
  data %>%
    mutate(
      advert = case_when(
        str_detect(!!sym(column_name), "informed_1") ~ "advert.1",
        str_detect(!!sym(column_name), "informed_2") ~ "advert.2",
        str_detect(!!sym(column_name), "informed_3") ~ "advert.3",
        str_detect(!!sym(column_name), "informed_4") ~ "advert.4",
        TRUE ~ NA_character_
      ),
      version = case_when(
        str_detect(!!sym(column_name), "imprint") ~ 1,
        TRUE ~ 0
      )
    ) 
}

PG_long <- in_advert_version(PG_long, "political_goal")
informed2_long <- in_advert_version(informed2_long, "informed2")
informed3_long <- in_advert_version(informed3_long, "informed3")
informed4_long <- in_advert_version(informed4_long, "informed4")

#the code below creates a function that filters out redundant rows, leaving 4 for each participant

clean_NA <- function(df) {
  # Identify the column(s) ending with '_value'
  value_cols <- names(df)[grepl("_value$", names(df))]
  
  # Ensure there is at least one column ending with '_value'
  if (length(value_cols) > 0) {
    df <- df %>%
      filter(!is.na(.[[value_cols]])) %>%
      distinct(id, advert, .keep_all = TRUE)
  }
  
  return(df)
}

#apply this function to all dataframes, specified through thier shared name of '_long' at the end of each df

df_names <- ls(pattern = "_long$")
df_list <- mget(df_names, envir = .GlobalEnv)

for (name in names(df_list)) {
  assign(name, clean_NA(get(name)), envir = .GlobalEnv)
}

#merge the dataframes back together by matching advert, participant id and version

rm_list <- list(PK1_long, PK2_long, PK3_long, PK4_long, PG_long, informed2_long, informed3_long, informed4_long, agree_long, trustworthy_long, accurate_long, believe_long, factual_long)

merged_rm <- reduce(rm_list, full_join, by = c("id", "advert", "version", "Advert.1", "Advert.2", "Advert.3", "Advert.4"))

#changing order of columns

merged_rm <- merged_rm %>%
  select(id, Advert.1, Advert.2, Advert.3, Advert.4, advert, version, everything())

#delete the variable columns e.g., 'PK1', 'informed2'

repeated_measures <- merged_rm %>%
  select(-c(PK1, PK2, PK3, PK4, political_goal, informed2, informed3, informed4, agree, trustworthy, believable, accurate, factual))

The code chunk below mean scores the persuasion knowledge items and the informed items. These are not the only scales that will be mean scored, but they are the only mean-scored items in the repeated measures part of the experiment (post-advert questions). Mean scoring of EPE and political trust items occur in a later section.

#convert to numerical

repeated_measures <- repeated_measures %>%
  convert_to_numeric(c("PG_value", "agree_value", "trustworthy_value", "believable_value", "accurate_value", "factual_value"))

#mean score

repeated_measures <- repeated_measures %>%
  rowwise() %>%
  mutate(PK = mean(c(PK1_value, PK2_value, PK3_value, PK4_value)))

repeated_measures <- repeated_measures %>%
  rowwise() %>%
  mutate(informed = mean(c(informed2_value, informed3_value, informed4_value)))

repeated_measures <- repeated_measures %>%
  rowwise() %>%
  mutate(PPI = mean(c(PK2_value, PK4_value)))

repeated_measures <- repeated_measures %>%
  rowwise() %>%
  mutate(PA = mean(c(PK1_value, PK3_value)))

repeated_measures <- repeated_measures %>%
  rowwise() %>%
  mutate(Credibility = mean(c(trustworthy_value, believable_value, accurate_value, factual_value)))

#changing the order of columns

repeated_measures <- repeated_measures %>%
  select(id, Advert.1, Advert.2, Advert.3, Advert.4, advert, version, PK, PPI, PA, informed, PG_value, agree_value, Credibility, trustworthy_value, believable_value, accurate_value, factual_value, everything())

Merged repeated measures data frame

The code below will now merge relevant variables from outside the repeated measures part of the experiment with this dataframe e.g., training condition, demographic variables and recall measures.

Variable descriptions for those with unclear names: - useful_rank_1 = where ‘voters’ were ranked by participants - SM_frequency_1 = how often participants use Facebook

#creating a new df with relevant variables e.g., controls for models

control_measures <- data %>%
  select(id, Training.condition, recall_num, recall_name, recall_correct, CSC, BBA, SFI, CBB, FF, TPM, VFP, AT, reg_know, useful_rank_1, political_interest, SM_use, SM_frequency_1, partyID, age_sample, gender, education, Ethnicity.simplified)

#matching id number with the repeated measures dataframe so these variables are repeated across rows

imprint_df <- repeated_measures %>%
  left_join(control_measures, by = "id")

#changing the order of columns

imprint_df <- imprint_df %>%
  select(id, Advert.1, Advert.2, Advert.3, Advert.4, Training.condition, advert, version, PK, PPI, PA, informed, PG_value, agree_value, Credibility, trustworthy_value, believable_value, accurate_value, factual_value, recall_num, recall_correct, CSC, BBA, SFI, CBB, FF, TPM, VFP, AT, political_interest, reg_know, SM_use, SM_frequency_1, partyID, age_sample, gender, education, Ethnicity.simplified, everything())

The code below conducts the following transformations to the variables so they are ready to be analysed:

  • Transformed to a factor: version, advert
  • Transformed to a numerical variable: PG_value, agree_value, trustworthy_value, believe_value, accurate_value, factual_value
#functions created in earlier section

imprint_df <- imprint_df %>%
  convert_to_factor(c("version", "advert"))

Independent measures data frame

Another aspect of the analysis will only require one row per participant, such as when testing the effect of the training condition on various outcomes e.g., confidence in regulation or epistemic political efficacy.

training_df <- data %>%
  select(id, Training.condition, Advert.1, Advert.2, Advert.3, Advert.4, election_reg, recall_num, recall_correct, name_correct, name_incorrect, CSC, BBA, SFI, CBB, FF, TPM, VFP, AT, starts_with("useful_rank"), reg_know, starts_with("EPE"), starts_with("general_confidence"), starts_with("institution_trust"), democracy, political_interest, external_efficacy, internal_efficacy, SM_use, starts_with("SM_frequency"), partyID, age_sample, gender, education, Ethnicity.simplified)

#Mean scoring EPE

training_df <- training_df %>%
  rowwise() %>%
  mutate(EPE_mean = mean(c(EPE_1, EPE_2, EPE_3, EPE_4)))

#Mean scoring trust, mistrust and cynicism

training_df <- training_df %>%
  rowwise() %>%
  mutate(political_trust = mean(c(general_confidence_1, general_confidence_2, general_confidence_3)))

training_df <- training_df %>%
  rowwise() %>%
  mutate(political_mistrust = mean(c(general_confidence_4, general_confidence_5, general_confidence_6)))

training_df <- training_df %>%
  rowwise() %>%
  mutate(political_cynicism = mean(c(general_confidence_7, general_confidence_8, general_confidence_9)))

Cleaning up the R environment

rm(list=setdiff(ls(), c("data", "imprint_df", "training_df")))

This document relies on the correct data frames having been formed from the code above which can be viewed under ‘details’. This information is also stored in a separate R Markdown document ‘datawrangling_code.Rmd’. The correct data frames used in the following analyses are called: imprint_df and training_df. Almost all hypotheses are tested using the former dataframe, which includes four rows for each participant to capture the repeated measures part of the experiment. Some hypotheses are tested using the latter dataframe, which includes only one row per participant.

Pre-registered research questions

Research theme 1: the effect of viewing a digital imprint on subsequent evaluations

  • Research question 1: Does the presence of a digital imprint increase citizens knowledge about the source of digital campaign material?

Research theme 2: the effect of being informed about the purpose of digital imprints on subsequent evaluations

  • Research question 2: How does being informed about the purpose of digital imprints affect citizens’ knowledge of the source of campaign material?
  • Research question 3: How does being informed about the purpose of digital imprints affect citizens’ perceptions of the trustworthiness of such material?
  • Research question 4: How does being informed about the purpose of a digital imprint affect citizen views on the sufficiency of current regulatory oversight?

Pre-registered hypotheses

Research question 1:

  • H1a: Digital imprints will increase respondents knowledge about the source of a piece of digital campaign material, with regards to the campaigners’ political and persuasive intent.
  • H1b: The presence of a digital imprint will not increase respondent’s memory of the names of campaigners whose post they viewed.
  • H1c: The presence of a digital imprint will not increase respondent’s perception that they are more informed about the source of campaign material.

Research question 2:

  • H2a: Those who are informed about the purpose of digital imprints will be more likely to correctly recall the names of the campaigners.
  • H2b: Those who are informed about the purpose of digital imprints will perceive themselves as informed about the source of a piece of material if and only if a digital imprint is present.

Research question 3:

  • H3: Those who are informed about the purpose of digital imprints will perceive campaign content as more trustworthy if and only if a digital imprint is present with the content.

Research question 4:

  • H4: Those who are informed about the purpose of an imprint are more likely to perceive campaign laws as sufficient compared to those who are not informed about the purpose.

Important note: pre-registration deviations in final paper

In the final analysis, some changes were made to the order of the hypotheses (please note that in content these stayed the same), and the analysis script that was pre-registered (the original can be viewed in the ‘rawdata_with_wranglecode’ folder of the github repository).

These changes did not reflect any substantial altering of the theorised associations between our variables, as reflected in how all hypotheses stayed the same. However, we altered the order of presentation for our hypotheses to ensure the most efficient reporting of the outcomes. This document includes the final version of all statistical models reported in the main paper.

If you would like to run the original pre-registered analysis script for comparison please follow these steps:

  • Locate the ‘rawdata_with_wranglecode’ folder in the repo
  • Ensure you have ‘main_data.csv’ saved in the same R project
  • Download the ‘preregistered_analysis_code.rmd’ script
  • Check you have packages installed (there are two sets, one for creating the data structure and another for the analysis packages)
  • Knit the script for ‘preregistered_analysis_code.rmd’

This should create an html document that can be scrolled through to view the original models, including all the assumptions for them (under the ‘details’ tabs)

R packages: visualisation and analysis

library(lme4)
## Loading required package: Matrix
## 
## Attaching package: 'Matrix'
## The following objects are masked from 'package:tidyr':
## 
##     expand, pack, unpack
library(lmerTest)
## 
## Attaching package: 'lmerTest'
## The following object is masked from 'package:lme4':
## 
##     lmer
## The following object is masked from 'package:stats':
## 
##     step
library(Matrix)
library(sjPlot)
library(ggplot2)
library(ggeffects)
library(performance)
library(see)
library(patchwork)
library(knitr)
library(kableExtra)
## 
## Attaching package: 'kableExtra'
## The following object is masked from 'package:dplyr':
## 
##     group_rows
library(broom)
library(broom.mixed)
library(htmltools)
library(rlang)
## 
## Attaching package: 'rlang'
## The following objects are masked from 'package:purrr':
## 
##     %@%, flatten, flatten_chr, flatten_dbl, flatten_int, flatten_lgl,
##     flatten_raw, invoke, splice
library(psych)
## 
## Attaching package: 'psych'
## The following objects are masked from 'package:ggplot2':
## 
##     %+%, alpha
library(lattice)
library(afex)
## ************
## Welcome to afex. For support visit: http://afex.singmann.science/
## - Functions for ANOVAs: aov_car(), aov_ez(), and aov_4()
## - Methods for calculating p-values with mixed(): 'S', 'KR', 'LRT', and 'PB'
## - 'afex_aov' and 'mixed' objects can be passed to emmeans() for follow-up tests
## - Get and set global package options with: afex_options()
## - Set sum-to-zero contrasts globally: set_sum_contrasts()
## - For example analyses see: browseVignettes("afex")
## ************
## 
## Attaching package: 'afex'
## The following object is masked from 'package:lme4':
## 
##     lmer
library(stats)
library(lavaan)
## This is lavaan 0.6-19
## lavaan is FREE software! Please report any bugs.
## 
## Attaching package: 'lavaan'
## The following object is masked from 'package:psych':
## 
##     cor2cov
library(lavaanPlot)
library(semTools)
## 
## ###############################################################################
## This is semTools 0.5-7
## All users of R (or SEM) are invited to submit functions or ideas for functions.
## ###############################################################################
## 
## Attaching package: 'semTools'
## The following objects are masked from 'package:psych':
## 
##     reliability, skew
library(ordinal)
## 
## Attaching package: 'ordinal'
## The following object is masked from 'package:dplyr':
## 
##     slice
library(MASS)
## 
## Attaching package: 'MASS'
## The following object is masked from 'package:patchwork':
## 
##     area
## The following object is masked from 'package:dplyr':
## 
##     select
library(brant)

Above are the R packages used for analysis and visualisation. Please ensure you have these installed if you wish to knit the script.

Pre-analysis Checks

Scale validity and reliability

The underlying structures of the following scales are checked using a CFA from the package Lavaan and semTools following the advice from Flora (2020).

  • Persuasion knowledge
  • Credibility
  • Perceived self-informedness

In assessing convergent validity, the guiding question is: Do these items appear to represent one latent construct? We used CFA factor loadings to assess this. Model fit indices are also important to check (Hair et al. 2014), however some model fit indices could not be investigated as there were not enough degrees of freedom in the model to test them (at least 4 items are needed). The models for perceived self-informedness, for example, were just-identified models. We therefore rely primarily on factor loadings.

In assessing internal reliability, the guiding question is: Do the items tend to be answered in a consistent way when responded to together? A high alpha total suggests that the items reflect a common underlying construct and can therefore be combined into a composite score.

As we had so few items for each measure, we did not consider the assumption of tau-equivelence that underscores alpha to be an issue.

Because the focus is on within-instance consistency (i.e., how participants respond to each set of items as a unit) the use of a repeated measures design and clustering (e.g., participant ID or stimulus type) does not interfere with this assessment.

## lavaan 0.6-19 ended normally after 15 iterations
## 
##   Estimator                                         ML
##   Optimization method                           NLMINB
##   Number of model parameters                         9
## 
##   Number of observations                          5288
##   Number of missing patterns                         1
## 
## Model Test User Model:
##                                               Standard      Scaled
##   Test Statistic                                 0.000       0.000
##   Degrees of freedom                                 0           0
## 
## Model Test Baseline Model:
## 
##   Test statistic                              8975.107    3563.660
##   Degrees of freedom                                 3           3
##   P-value                                        0.000       0.000
##   Scaling correction factor                                  2.519
## 
## User Model versus Baseline Model:
## 
##   Comparative Fit Index (CFI)                    1.000       1.000
##   Tucker-Lewis Index (TLI)                       1.000       1.000
##                                                                   
##   Robust Comparative Fit Index (CFI)                         1.000
##   Robust Tucker-Lewis Index (TLI)                            1.000
## 
## Loglikelihood and Information Criteria:
## 
##   Loglikelihood user model (H0)             -26797.163  -26797.163
##   Loglikelihood unrestricted model (H1)     -26797.163  -26797.163
##                                                                   
##   Akaike (AIC)                               53612.325   53612.325
##   Bayesian (BIC)                             53671.484   53671.484
##   Sample-size adjusted Bayesian (SABIC)      53642.885   53642.885
## 
## Root Mean Square Error of Approximation:
## 
##   RMSEA                                          0.000          NA
##   90 Percent confidence interval - lower         0.000          NA
##   90 Percent confidence interval - upper         0.000          NA
##   P-value H_0: RMSEA <= 0.050                       NA          NA
##   P-value H_0: RMSEA >= 0.080                       NA          NA
##                                                                   
##   Robust RMSEA                                               0.000
##   90 Percent confidence interval - lower                     0.000
##   90 Percent confidence interval - upper                     0.000
##   P-value H_0: Robust RMSEA <= 0.050                            NA
##   P-value H_0: Robust RMSEA >= 0.080                            NA
## 
## Standardized Root Mean Square Residual:
## 
##   SRMR                                           0.000       0.000
## 
## Parameter Estimates:
## 
##   Standard errors                             Sandwich
##   Information bread                           Observed
##   Observed information based on                Hessian
## 
## Latent Variables:
##                    Estimate  Std.Err  z-value  P(>|z|)
##   informed =~                                         
##     informed2_valu    1.596    0.016   98.777    0.000
##     informed3_valu    1.432    0.021   69.153    0.000
##     informed4_valu    1.399    0.018   78.168    0.000
## 
## Intercepts:
##                    Estimate  Std.Err  z-value  P(>|z|)
##    .informed2_valu    4.203    0.024  174.168    0.000
##    .informed3_valu    3.992    0.025  160.015    0.000
##    .informed4_valu    4.422    0.023  194.801    0.000
## 
## Variances:
##                    Estimate  Std.Err  z-value  P(>|z|)
##    .informed2_valu    0.531    0.033   16.110    0.000
##    .informed3_valu    1.242    0.052   23.896    0.000
##    .informed4_valu    0.767    0.033   23.133    0.000
##     informed          1.000
##         informed
## alpha  0.8837737
## omega  0.8852859
## omega2 0.8852859
## omega3 0.8852859
## avevar 0.7207662
## lavaan 0.6-19 ended normally after 21 iterations
## 
##   Estimator                                         ML
##   Optimization method                           NLMINB
##   Number of model parameters                        12
## 
##   Number of observations                          5288
##   Number of missing patterns                         1
## 
## Model Test User Model:
##                                               Standard      Scaled
##   Test Statistic                              1790.370     825.795
##   Degrees of freedom                                 2           2
##   P-value (Chi-square)                           0.000       0.000
##   Scaling correction factor                                  2.168
##     Yuan-Bentler correction (Mplus variant)                       
## 
## Model Test Baseline Model:
## 
##   Test statistic                              9349.940    5135.159
##   Degrees of freedom                                 6           6
##   P-value                                        0.000       0.000
##   Scaling correction factor                                  1.821
## 
## User Model versus Baseline Model:
## 
##   Comparative Fit Index (CFI)                    0.809       0.839
##   Tucker-Lewis Index (TLI)                       0.426       0.518
##                                                                   
##   Robust Comparative Fit Index (CFI)                         0.809
##   Robust Tucker-Lewis Index (TLI)                            0.426
## 
## Loglikelihood and Information Criteria:
## 
##   Loglikelihood user model (H0)             -32360.019  -32360.019
##   Scaling correction factor                                  1.504
##       for the MLR correction                                      
##   Loglikelihood unrestricted model (H1)     -31464.834  -31464.834
##   Scaling correction factor                                  1.599
##       for the MLR correction                                      
##                                                                   
##   Akaike (AIC)                               64744.039   64744.039
##   Bayesian (BIC)                             64822.917   64822.917
##   Sample-size adjusted Bayesian (SABIC)      64784.785   64784.785
## 
## Root Mean Square Error of Approximation:
## 
##   RMSEA                                          0.411       0.279
##   90 Percent confidence interval - lower         0.395       0.268
##   90 Percent confidence interval - upper         0.427       0.290
##   P-value H_0: RMSEA <= 0.050                    0.000       0.000
##   P-value H_0: RMSEA >= 0.080                    1.000       1.000
##                                                                   
##   Robust RMSEA                                               0.411
##   90 Percent confidence interval - lower                     0.392
##   90 Percent confidence interval - upper                     0.431
##   P-value H_0: Robust RMSEA <= 0.050                         0.000
##   P-value H_0: Robust RMSEA >= 0.080                         1.000
## 
## Standardized Root Mean Square Residual:
## 
##   SRMR                                           0.138       0.138
## 
## Parameter Estimates:
## 
##   Standard errors                             Sandwich
##   Information bread                           Observed
##   Observed information based on                Hessian
## 
## Latent Variables:
##                    Estimate  Std.Err  z-value  P(>|z|)
##   pk =~                                               
##     PK1_value         1.777    0.021   84.280    0.000
##     PK2_value         0.140    0.013   10.617    0.000
##     PK3_value         1.674    0.022   75.223    0.000
##     PK4_value         0.231    0.015   15.380    0.000
## 
## Intercepts:
##                    Estimate  Std.Err  z-value  P(>|z|)
##    .PK1_value         3.649    0.026  140.892    0.000
##    .PK2_value         6.226    0.012  499.385    0.000
##    .PK3_value         3.478    0.025  137.980    0.000
##    .PK4_value         6.058    0.014  432.801    0.000
## 
## Variances:
##                    Estimate  Std.Err  z-value  P(>|z|)
##    .PK1_value         0.389    0.063    6.210    0.000
##    .PK2_value         0.802    0.030   27.185    0.000
##    .PK3_value         0.557    0.059    9.476    0.000
##    .PK4_value         0.983    0.032   30.492    0.000
##     pk                1.000
##               pk
## alpha  0.6908958
## omega  0.8425706
## omega2 0.8425706
## omega3 0.8033694
## avevar 0.6884679
##          lhs op       rhs est.std    se       z pvalue ci.lower ci.upper
## 1         pk =~ PK1_value   0.944 0.009 100.593      0    0.925    0.962
## 2         pk =~ PK2_value   0.155 0.014  11.387      0    0.128    0.181
## 3         pk =~ PK3_value   0.913 0.010  94.269      0    0.894    0.932
## 4         pk =~ PK4_value   0.227 0.014  16.595      0    0.200    0.254
## 5  PK1_value ~~ PK1_value   0.110 0.018   6.197      0    0.075    0.144
## 6  PK2_value ~~ PK2_value   0.976 0.004 231.990      0    0.968    0.984
## 7  PK3_value ~~ PK3_value   0.166 0.018   9.359      0    0.131    0.200
## 8  PK4_value ~~ PK4_value   0.948 0.006 152.601      0    0.936    0.961
## 9         pk ~~        pk   1.000 0.000      NA     NA    1.000    1.000
## 10 PK1_value ~1             1.937 0.016 123.703      0    1.907    1.968
## 11 PK2_value ~1             6.867 0.139  49.526      0    6.596    7.139
## 12 PK3_value ~1             1.897 0.015 126.469      0    1.868    1.927
## 13 PK4_value ~1             5.952 0.109  54.843      0    5.739    6.164
## 14        pk ~1             0.000 0.000      NA     NA    0.000    0.000
##               pk
## alpha  0.9258648
## omega  0.9259989
## omega2 0.9259989
## omega3 0.9259989
## avevar 0.8622127
## lavaan 0.6-19 ended normally after 18 iterations
## 
##   Estimator                                         ML
##   Optimization method                           NLMINB
##   Number of model parameters                        12
## 
##   Number of observations                          5288
##   Number of missing patterns                         1
## 
## Model Test User Model:
##                                               Standard      Scaled
##   Test Statistic                               255.957     208.837
##   Degrees of freedom                                 2           2
##   P-value (Chi-square)                           0.000       0.000
##   Scaling correction factor                                  1.226
##     Yuan-Bentler correction (Mplus variant)                       
## 
## Model Test Baseline Model:
## 
##   Test statistic                             15471.247    8244.147
##   Degrees of freedom                                 6           6
##   P-value                                        0.000       0.000
##   Scaling correction factor                                  1.877
## 
## User Model versus Baseline Model:
## 
##   Comparative Fit Index (CFI)                    0.984       0.975
##   Tucker-Lewis Index (TLI)                       0.951       0.925
##                                                                   
##   Robust Comparative Fit Index (CFI)                         0.984
##   Robust Tucker-Lewis Index (TLI)                            0.951
## 
## Loglikelihood and Information Criteria:
## 
##   Loglikelihood user model (H0)             -31908.509  -31908.509
##   Scaling correction factor                                  1.308
##       for the MLR correction                                      
##   Loglikelihood unrestricted model (H1)     -31780.531  -31780.531
##   Scaling correction factor                                  1.296
##       for the MLR correction                                      
##                                                                   
##   Akaike (AIC)                               63841.019   63841.019
##   Bayesian (BIC)                             63919.897   63919.897
##   Sample-size adjusted Bayesian (SABIC)      63881.765   63881.765
## 
## Root Mean Square Error of Approximation:
## 
##   RMSEA                                          0.155       0.140
##   90 Percent confidence interval - lower         0.139       0.126
##   90 Percent confidence interval - upper         0.171       0.155
##   P-value H_0: RMSEA <= 0.050                    0.000       0.000
##   P-value H_0: RMSEA >= 0.080                    1.000       1.000
##                                                                   
##   Robust RMSEA                                               0.155
##   90 Percent confidence interval - lower                     0.138
##   90 Percent confidence interval - upper                     0.173
##   P-value H_0: Robust RMSEA <= 0.050                         0.000
##   P-value H_0: Robust RMSEA >= 0.080                         1.000
## 
## Standardized Root Mean Square Residual:
## 
##   SRMR                                           0.019       0.019
## 
## Parameter Estimates:
## 
##   Standard errors                             Sandwich
##   Information bread                           Observed
##   Observed information based on                Hessian
## 
## Latent Variables:
##                    Estimate  Std.Err  z-value  P(>|z|)
##   credible =~                                         
##     trustworthy_vl    1.292    0.015   84.351    0.000
##     believable_val    1.389    0.016   88.349    0.000
##     accurate_value    1.343    0.015   88.762    0.000
##     factual_value     1.234    0.021   57.537    0.000
## 
## Intercepts:
##                    Estimate  Std.Err  z-value  P(>|z|)
##    .trustworthy_vl    3.676    0.020  179.817    0.000
##    .believable_val    4.496    0.021  210.929    0.000
##    .accurate_value    4.211    0.020  210.187    0.000
##    .factual_value     3.263    0.025  132.099    0.000
## 
## Variances:
##                    Estimate  Std.Err  z-value  P(>|z|)
##    .trustworthy_vl    0.541    0.018   30.516    0.000
##    .believable_val    0.474    0.021   22.616    0.000
##    .accurate_value    0.318    0.018   18.066    0.000
##    .factual_value     1.704    0.048   35.442    0.000
##     credible          1.000
##         credible
## alpha  0.9012809
## omega  0.9010190
## omega2 0.9010190
## omega3 0.8992711
## avevar 0.6951328
##                  lhs op               rhs est.std    se       z pvalue ci.lower
## 1           credible =~ trustworthy_value   0.869 0.005 175.592      0    0.859
## 2           credible =~  believable_value   0.896 0.005 179.385      0    0.886
## 3           credible =~    accurate_value   0.922 0.005 198.129      0    0.913
## 4           credible =~     factual_value   0.687 0.010  67.281      0    0.667
## 5  trustworthy_value ~~ trustworthy_value   0.245 0.009  28.489      0    0.228
## 6   believable_value ~~  believable_value   0.197 0.009  22.031      0    0.180
## 7     accurate_value ~~    accurate_value   0.150 0.009  17.479      0    0.133
## 8      factual_value ~~     factual_value   0.528 0.014  37.618      0    0.500
## 9           credible ~~          credible   1.000 0.000      NA     NA    1.000
## 10 trustworthy_value ~1                     2.473 0.025  98.141      0    2.423
## 11  believable_value ~1                     2.901 0.032  90.817      0    2.838
## 12    accurate_value ~1                     2.890 0.031  92.432      0    2.829
## 13     factual_value ~1                     1.817 0.015 120.193      0    1.787
## 14          credible ~1                     0.000 0.000      NA     NA    0.000
##    ci.upper
## 1     0.879
## 2     0.906
## 3     0.931
## 4     0.707
## 5     0.262
## 6     0.215
## 7     0.167
## 8     0.555
## 9     1.000
## 10    2.522
## 11    2.963
## 12    2.952
## 13    1.846
## 14    0.000

Sample

A representative sample option was used when collecting data from Prolific, matching UK census data for age, gender and ethnicity. The following table shows the make up of the sample for these demographics as well as education and political party identification, after exclusions.

Sample Breakdown
Sample Breakdown

Univariate statistics

Below provides descriptive statistics for the key predictor and outcome variables. The first table lists the continuous measures in the repeated measures part of the experiment, and the second shows confidence in regulation, which was only measured once. As can be seen, perceived political goal was heavily skewed. This suggested the political nature of each post was easy for participants to infer.

Summary Statistics for repeated measures variables
Variable Mean Median SD Min Max Q1 Q3
Perceived informedness 4.21 4.33 1.57 1 7 3 5.67
Perceived persuasive intent 6.14 6.00 0.85 1 7 6 7.00
Perceived advertisement 3.56 3.50 1.79 1 7 2 5.00
Perceived political goal 6.03 6.00 1.13 1 7 6 7.00
Credibility 3.91 4.00 1.39 1 7 3 4.75
Summary Statistics for independent measure variables
Variable Mean Median SD Min Max Q1 Q3
Confidence in regulation 2.95 3 1.43 1 7 2 4

Below shows this information split by the two conditions: training and version viewed.

Univariate Histograms

Below then shows the distribution of each variable as a histogram.

PG Value Histogram
PK Histogram
Informed Histogram
Agree Value Histogram
Trustworthy Value Histogram
Believable Value Histogram
Factual Value Histogram
Accurate Value Histogram
Election Regulation Histogram

Bivariate Statistics

Below shows the percentage recall for each of the campaigner names. As can be seen, there is variation between the names, with ‘Speak Freely Inc’ resulting in the lowest recall, and ‘Campaign for a Better Britain’ the highest. This suggests some names were either more memorable than others, or were more visually obvious on the page. This variation allows us to investigate if digital imprints consistently improve recall regardless of these overall differences in recall between names. This helps uncover the effectiveness of digital imprints across different formats in increasing citizen awareness of which campaigners are potentially targeting them during an election. Assessing each advert separately, which will be included as a supplementary analysis to hypothesis 2a, helps decipher how obvious digital imprints need to be to be effective in increasing recall.

Percentage of Participants Who Recalled Each Name
Recall.Percentage No_recall
Common Sense Collective 63.09 36.91
Breaking Barriers Alliance 46.22 53.78
Speak Freely Inc 37.67 62.33
Campaign for a Better Britain 71.41 28.59
Total 54.60 45.40 
## # A tibble: 2 × 9
##   Training.condition CSC_Recall CSC_No_Recall BBA_Recall BBA_No_Recall
##   <fct>                   <dbl>         <dbl>      <dbl>         <dbl>
## 1 0                        60.1          39.9       43.4          56.6
## 2 1                        66.1          33.9       49.0          51.0
## # ℹ 4 more variables: SFI_Recall <dbl>, SFI_No_Recall <dbl>, CBB_Recall <dbl>,
## #   CBB_No_Recall <dbl>

Manipulation check

The manipulation check was to check if the training condition successfully increased the attention paid to the disclosures during exposure to the experimental conditions. The two measures in the training_df dataframe of interest are:

  • recall_num: the number of digital imprints recalled (how many did you view on the four materials)
  • recall_correct: whether the correct answer of ‘two’ was selected

It can be seen a majority of participants correctly selected two.

##           
##               0    1
##   Not sure 21.9 17.0
##   0        13.4 13.6
##   1        20.8 18.1
##   2        23.8 26.5
##   3        15.5 18.6
##   4         4.7  6.2
##            
##               0   1
##   incorrect 502 487
##   correct   157 176
  recall_correct
Predictors Odds Ratios CI p
(Intercept) 0.31 0.26 – 0.37 <0.001
Training condition [1] 1.16 0.90 – 1.48 0.254
Observations 1322
R2 Tjur 0.001 
## # weights:  18 (10 variable)
## initial  value 2368.706018 
## iter  10 value 2267.640271
## final  value 2261.184048 
## converged
  recall_num
Predictors Odds Ratios CI p Response
(Intercept) 0.61 0.47 – 0.80 <0.001 0
Training condition [1] 1.30 0.89 – 1.91 0.176 0
(Intercept) 0.95 0.75 – 1.20 0.676 1
Training condition [1] 1.12 0.79 – 1.58 0.535 1
(Intercept) 1.09 0.87 – 1.37 0.454 2
Training condition [1] 1.43 1.03 – 1.98 0.033 2
(Intercept) 0.71 0.55 – 0.91 0.008 3
Training condition [1] 1.54 1.07 – 2.20 0.019 3
(Intercept) 0.22 0.15 – 0.32 <0.001 4
Training condition [1] 1.69 0.99 – 2.86 0.053 4
Observations 1322
R2 / R2 adjusted 0.002 / 0.002
  Training.condition
Predictors Odds Ratios CI p
(Intercept) 0.34 0.02 – 2.79 0.359
partyID [Conservative] 3.19 0.40 – 65.23 0.319
partyID [Democratic
Unionist Party] 		1.76 0.05 – 68.77 0.740
partyID [Green Party] 2.79 0.34 – 57.56 0.381
partyID [I do not
identify with any
political party] 		3.42 0.43 – 69.63 0.291
partyID [Labour] 3.02 0.38 – 61.50 0.340
partyID [Liberal
Democrat] 		2.33 0.29 – 48.16 0.470
partyID [Other, please
specify___________] 		2.90 0.31 – 64.23 0.389
partyID [Plaid Cymru] 11.08 0.56 – 545.23 0.149
partyID [Reform UK] 3.17 0.38 – 66.43 0.331
partyID [Scottish
National Party (SNP)] 		2.22 0.25 – 47.83 0.510
partyID [Sinn Féin] 5.70 0.25 – 281.49 0.302
partyID [United Kingdom
Independence Party
(UKIP)] 		10.74 0.56 – 509.38 0.149
age sample 1.00 0.99 – 1.01 0.956
gender [Male] 1.02 0.82 – 1.27 0.857
gender [Non-binary /
third gender] 		0.54 0.06 – 3.67 0.541
gender [Prefer not to
say] 		0.72 0.14 – 3.31 0.665
education [Undergraduate
University (e.g. BA,
B.Sc, B.Ed)] 		1.00 0.75 – 1.34 0.993
education [A-level, or
equivalent] 		0.83 0.59 – 1.18 0.296
education [GCSE level, or
equivalent] 		0.88 0.59 – 1.30 0.509
education [Other, please
specify] 		0.73 0.24 – 2.15 0.567
education [No formal
qualifications] 		0.59 0.17 – 1.86 0.374
Ethnicity simplified
[Asian] 		0.87 0.57 – 1.34 0.525
Ethnicity simplified
[Black] 		1.39 0.74 – 2.68 0.312
Ethnicity simplified
[Mixed] 		1.21 0.51 – 2.96 0.668
Ethnicity simplified
[Other] 		2.06 0.90 – 5.11 0.100
Observations 1316
R2 Tjur 0.011

Knowledge of Regulation

Below shows how regulation knowledge was distributed across the whole sample, with the third option being the correct answer. This measure is not included in the pre-registered analysis, but is helpful to get a sense of how knowledgeable the sample was regarding regulatory law in the UK. Training and no training condition are shown separately. It can be seen, the training did not appear to impact the frequency of responses, with the highest count of participant in each condition identifying the correct response option across both conditions.

Usefullness Rankings

Below shows where in the ranking participants tended to rate ‘voters’ when asked who the digital imprint information was most useful for.

## `summarise()` has grouped output by 'Training.condition'. You can override
## using the `.groups` argument.

## `summarise()` has grouped output by 'Training.condition'. You can override
## using the `.groups` argument.

Persuasion knowledge hypotheses

Why random effects at all?

Random effects are essential to include because it is expected individual participants will evaluate the adverts from different baselines. Fitting a model that accounts for this variance in baseline assessments among participants recognises the individual differences in perceptions that occur in response patterns.

Random effects are also necessary for the adverts themselves, to acknowledge that each advert will also elicit different perceptions due to their content or nature. Capturing this variability reflects the reality that some adverts, by their design are more, for example, politically charged or persuasive than others, thereby starting from different evaluative baselines.

Outcome: Perceived persuasive intent PPI

Did the presence of a digital imprint with a piece of campaign material increase participant awareness that the material was trying to persuade them of a certain viewpoint?

Plot: raw data

## `geom_smooth()` using formula = 'y ~ x'

Pre-registration Deviations

Pre-registered model:

  • model 2 <- lmer(PPI ~ imprint viewed + (1|id) + (1|advert))

Reported model:

  • model 2 <- lmer(PPI ~ version + training + agreement + (1|participant id) + (1|material))

Main deviation:

  • The effect of training was included as, although not central to the hypotheses, it was recognised the training may have also influenced perceptions of the outcome that needed to be accounted for to ensure the effect of version was not overestimated
  • Agreement included as a control.

Model fit metrics: AIC

As with the political goal outcome, the same set of model structures were tested for their AIC values.

Theoretically feasible model structures:

  • Random intercepts: Outcome ~ 1 + version + training + agreement + (1|participant id) + (1|material)
  • Maximal random slopes model: Outcome ~ 1 + version + training + agreement + (1|participant id) + (1 + version|material)
  • Material:version intercepts model: Outcome ~ 1 + version + training + agreement + (1|participant id) + (1|material:version)
  • Fixed-effect interaction: Outcome ~ 1 + version*training + agreement + (1|participant id) + (1|material)

In this case, the final chosen model was structure 1. As can be seen below in the model variation comparisons, increasing model complexity with random slopes or the inclusion of interaction effects worsened the fit of the model when evaluated using AIC values. This strongly implies adding model complexity in the form of random slopes does not aid a valid estimation of effects in this case, likely due to the small effects of digital imprint version.

AIC Values and Variables: Persuasion Knowledge
Variations Variables AIC
Model 1 version + training + agreement + id (random intercept) + advert (random intercept) 12246.02
Model 2 version + training + agreement + id (random intercept) + (version & advert (random slope)) 12248.72
Model 3 version + training + agreement + id (random intercept) + (advert:version (random intercepts vary)) 12249.40
Model 4 version*training + agreement + id (random intercept) + advert (random intercept) 12252.82

Model outcomes: Table

Outcome: PPI
Term Coefficient Std. Error Lower CI Upper CI p-value
(Intercept) 6.300 0.082 6.097 6.502 0.000
Digital imprint viewed (ref: not viewed) |0.033 |0.018 |-0.002 |0.069 |0.066
Training (ref: no training) |0.064 |0.034 |-0.002 |0.129 |0.058
Agreement -0.046 0.008 -0.061 -0.030 0.000
Random Effect (id) 0.264
Random Effect (advert) 0.019
ICC 0.396
σ² 0.657
Marginal R² 0.008
Conditional R² 0.401

False Discovery Rate

As this model found support for rejecting the null regarding the effect of both viewing a digital imprint and receiving training on persuasion knowledge, the p-values were adjusted to account for the multiple comparisons made during this analysis and ensure the robustness of the findings.

Original and Adjusted p-Values for Fixed Effects
Fixed Effect Original P-Value Adjusted P-Value
(Intercept) 0.000 0.000
Digital imprint viewed (ref: not viewed) |0.066 |0.066
Training (ref: no training) |0.058 |0.066
Agreement 0.000 0.000

Plot: Model Predictions

Model Assumptions

The assumptions checked below are as follows:

  • normality of residuals
  • variance of residuals
  • normality of residuals for each of the random effects (id and advert)
Shapiro-Wilk Test Results for Normality
Variable W_Statistic P_Value
id_effects 0.955 0.000
advert_effects 0.777 0.067
Note:
A p-value > 0.05 indicates the data is likely to be normally distributed. A p-value ≤ 0.05 suggests the data deviates from normality.

CLMM

The previous assumption charts show severe violation regarding the normality of the residuals, due to the skew of the outcome variable.

It can be seen in the histogram this variable has a non-normal distribution, suggesting a non-parametric approach is more suitable. As the outcome increases in 0.5 increments, a cumulative link mixed-effects model (CLMM) is used to estimate the effects.

The use of brant to check the proportional odds assumption gives an error message due to the small number of observations in some value of ‘1’ which makes it difficult for the model to accurately compare these sets of observations. However, on eyeballing the graph that plots this assumption, it can be seen there is no clear violation (the lines do not cross).

## ---------------------------------------------------- 
## Test for     X2  df  probability 
## ---------------------------------------------------- 
## Omnibus          39.72   33  0.2
## version1     7.81    11  0.73
## Training.condition1  9.03    11  0.62
## agree_value      22.75   11  0.02
## ---------------------------------------------------- 
## 
## H0: Parallel Regression Assumption holds
Table 1: Fixed Effects for Ordinal Mixed-Effects Model
Odds_Ratio Estimate Std_Error Z_Value P_Value
1|1.5 0.000 -8.277 0.505 -16.406 < 0.001
1.5|2 0.000 -7.805 0.424 -18.422 < 0.001
2|2.5 0.001 -7.171 0.348 -20.608 < 0.001
2.5|3 0.001 -6.613 0.305 -21.704 < 0.001
3|3.5 0.002 -6.098 0.278 -21.898 < 0.001
3.5|4 0.005 -5.339 0.255 -20.939 < 0.001
4|4.5 0.013 -4.369 0.239 -18.252 < 0.001
4.5|5 0.025 -3.679 0.233 -15.762 < 0.001
5|5.5 0.068 -2.693 0.228 -11.789 < 0.001
5.5|6 0.169 -1.780 0.226 -7.877 < 0.001
6|6.5 1.327 0.283 0.224 1.264 0.206
6.5|7 4.290 1.456 0.225 6.468 < 0.001
version1 1.112 0.106 0.053 2.006 0.045
Training.condition1 1.227 0.204 0.109 1.873 0.061 
##               1|1.5               1.5|2               2|2.5               2.5|3 
##        3.487296e-60        2.457783e-75        8.185659e-94       1.303604e-103 
##               3|3.5               3.5|4               4|4.5               4.5|5 
##       3.817267e-105        1.110700e-96        4.659961e-74        9.923887e-56 
##               5|5.5               5.5|6               6|6.5               6.5|7 
##        6.891966e-32        4.678181e-15        2.063740e-01        1.261773e-10 
##            version1 Training.condition1 
##        5.233020e-02        6.582576e-02

Outcome: Perceived advertisement

Did the presence of a digital imprint with a piece of campaign material increase participant awareness that the material was an advertisement?

This model was not pre-registered, but would introduced on the back of construct validity checks supporting that the four items measuring PK were actually two seperate constructs.

Plot: raw data

## `geom_smooth()` using formula = 'y ~ x'

Pre-registration Deviations

Pre-registered model:

  • model 2 <- lmer(persuasion knowledge ~ imprint viewed + (1|id) + (1|advert))

Reported model:

  • model 2 <- lmer(persuasion knowledge ~ version + training + agreement + (1|participant id) + (1|material))

Main deviation:

  • The effect of training was included as, although not central to the hypotheses, it was recognised the training may have also influenced perceptions of the outcome that needed to be accounted for to ensure the effect of version was not overestimated
  • The effect of agreement was included for comparison with other models.

Model fit metrics: AIC

As with the political goal outcome, the same set of model structures were tested for their AIC values.

Theoretically feasible model structures:

  • Random intercepts: Outcome ~ 1 + version + training + agreement + (1|participant id) + (1|material)
  • Maximal random slopes model: Outcome ~ 1 + version + training + agreement + (1|participant id) + (1 + version|material)
  • Material:version intercepts model: Outcome ~ 1 + version + training + agreement + (1|participant id) + (1|material:version)
  • Fixed-effect interaction: Outcome ~ 1 + version*training + agreement + (1|participant id) + (1|material)

In this case, the final chosen model was structure 1. As can be seen below in the model variation comparisons, increasing model complexity with random slopes or the inclusion of interaction effects worsened the fit of the model when evaluated using AIC values. This strongly implies adding model complexity in the form of random slopes does not aid a valid estimation of effects in this case, likely due to the small effects of digital imprint version.

## boundary (singular) fit: see help('isSingular')
AIC Values and Variables: PA
Variations Variables AIC
Model 1 version + training + agreement + id (random intercept) + advert (random intercept) 19369.69
Model 2 version + training + agreement + id (random intercept) + (version & advert (random slope)) 19372.96
Model 3 version + training + agreement + id (random intercept) + (advert:version (random intercepts vary)) 19384.19
Model 4 version*training + agreement + id (random intercept) + advert (random intercept) 19374.15

Model outcomes: Table

Outcome: persuasion knowledge
Term Coefficient Std. Error Lower CI Upper CI p-value
(Intercept) 3.817 0.380 2.671 4.964 0.001
Digital imprint viewed (ref: not viewed) |0.162 |0.035 |0.093 |0.231 |0.000
Training (ref: no training) |0.230 |0.067 |0.099 |0.362 |0.001
Agreement -0.099 0.015 -0.130 -0.069 0.000
Random Effect (id) 1.075
Random Effect (advert) 0.549
ICC 0.498
σ² 1.280
Marginal R² 0.013
Conditional R² 0.504

False Discovery Rate

As this model found support for rejecting the null regarding the effect of both viewing a digital imprint and receiving training on persuasion knowledge, the p-values were adjusted to account for the multiple comparisons made during this analysis and ensure the robustness of the findings.

Original and Adjusted p-Values for Fixed Effects
Fixed Effect Original P-Value Adjusted P-Value
(Intercept) 0.001 0.001
Digital imprint viewed (ref: not viewed) |0.000 |0.000
Training (ref: no training) |0.001 |0.001
Agreement 0.000 0.000

Plot: Model Predictions

Model Assumptions

The assumptions checked below are as follows:

  • normality of residuals
  • variance of residuals
  • normality of residuals for each of the random effects (id and advert)
Shapiro-Wilk Test Results for Normality
Variable W_Statistic P_Value
id_effects 0.994 0.000
advert_effects 0.910 0.483
Note:
A p-value > 0.05 indicates the data is likely to be normally distributed. A p-value ≤ 0.05 suggests the data deviates from normality.

Perceived self informedness hypotheses

Outcome: Perceived Informedness

Did the presence of a digital imprint with a piece of campaign material increase participant’s perception that they had been informed about the source of the content?

Were trained participants more likely to correctly identify that they were less informed when a digital imprint was not present, and more informed when a digital imprint was present, compared to the group who received no training?

Plot: raw data

Decriptive bivariate statistics for percieved informedness
trained version n mean sd
0 0 1318 4.08 1.56
0 1 1318 4.29 1.54
1 0 1326 4.01 1.60
1 1 1326 4.44 1.54

Pre-registration deviations

Model fit metrics: AIC

For this model, the inclusion of an interaction term was central to hypotheses 3b. For this reason, only alternates in random effects structures are compared.

## boundary (singular) fit: see help('isSingular')
AIC Values and Variables: Perceived informedness
Variations Variables AIC
Structure 1 version*training + agreement + id (random intercept) + advert (random intercept) 19011.94
Structure 2 version*training + agreement + id (random intercept) + (version & advert (random slope)) 19011.49
Structure 3 version*training + agreement + id (random intercept) + (advert:version (random intercepts vary)) 19018.28

Model outcomes: table

Outcome: perceived informedness
Term Coefficient Std. Error Lower CI Upper CI p-value
(Intercept) 3.501 0.214 2.913 4.090 0.000
Training.condition1 -0.067 0.066 -0.197 0.064 0.316
version1 0.212 0.051 0.113 0.311 0.000
Agreement 0.127 0.015 0.098 0.156 0.000
Training.condition1:version1 0.222 0.071 0.082 0.362 0.002
Random Effect (id) 0.616
Random Effect (advert) 0.155
ICC 0.314
σ² 1.299
Marginal R² 0.026
Conditional R² 0.331

False discovery rate

Original and Adjusted p-Values for Fixed Effects
Fixed Effect Original P-Value Adjusted P-Value
(Intercept) 0.000 0.000
Training.condition1 0.316 0.316
version1 0.000 0.000
Agreement 0.000 0.000
Training.condition1:version1 0.002 0.002

Model outcomes: no interaction

Outcome: perceived informedness
Term Coefficient Std. Error Lower CI Upper CI p-value
(Intercept) 3.447 0.213 2.856 4.037 0.000
Training.condition1 0.044 0.056 -0.066 0.154 0.429
version1 0.323 0.036 0.253 0.393 0.000
Agreement 0.127 0.015 0.098 0.156 0.000
Random Effect (id) 0.615
Random Effect (advert) 0.156
ICC 0.313
σ² 1.300
Marginal R² 0.025
Conditional R² 0.330

False discovery rate

Original and Adjusted p-Values for Fixed Effects
Fixed Effect Original P-Value Adjusted P-Value
(Intercept) 0.000 0.000
Training.condition1 0.429 0.429
version1 0.000 0.000
Agreement 0.000 0.000

Plot: model predictions

## Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
## ℹ Please use `linewidth` instead.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.

Model assumptions

The following assumptions are checked:

  • Normality of residuals
  • Variance of residuals
  • Normality of residuals within the random effects

There are no key assumption violations, supporting that the model predictions are reliable.

Shapiro-Wilk Test Results for Normality
Variable W_Statistic P_Value
id_effects 0.998 0.214
advert_effects 0.930 0.592
Note:
A p-value > 0.05 indicates the data is likely to be normally distributed. A p-value ≤ 0.05 suggests the data deviates from normality.

Credibility hypotheses

outcome: credibility

Did those informed about the purpose of imprints use their absence/presence to evaluate the trustworthiness/credibility of the posts?

Plot: raw data

Below shows the raw data distributions split by both experimental manipulations (training and version) on the two key outcomes of interest.

Pre-registration Deviations

Pre-registered models:

Models reported in paper:

  • model 5 <- lmer(credibility ~ Training.condition + version + agree_value + Training.condition*version + (1|id) + (1|advert))

Reason for divergence:

  • The variables loaded well onto an aggregated score which, following previous research, are used to measure perceptions of credibility.

Model fit metrics: AIC

For this model, the inclusion of an interaction term was central to hypotheses 4b. For this reason, only alternates in random effects structures are compared.

## boundary (singular) fit: see help('isSingular')
AIC Values and Variables: Credibility
Variations Variables AIC
Structure 1 version + training + agreement + id (random intercept) + advert (random intercept) 13120.21
Structure 2 version*training + agreement + id (random intercept) + advert (random intercept) 13125.86
Structure 3 version*training + agreement + id (random intercept) + (version & advert (random slope)) 13125.61
Structure 4 version*training + agreement + id (random intercept) + (advert:version (random intercepts vary)) 13133.78 
## Contrasts set to contr.sum for the following variables: Training.condition, version, advert
## Numerical variables NOT centered on 0: agree_value
## If in interactions, interpretation of lower order (e.g., main) effects difficult.
## REML argument to lmer() set to FALSE for method = 'PB' or 'LRT'
## Mixed Model Anova Table (Type 3 tests, LRT-method)
## 
## Model: Credibility ~ Training.condition + version + agree_value + Training.condition:version + 
## Model:     (1 | id) + (1 | advert)
## Data: imprint_df
## Df full model: 8
##                       Effect df       Chisq p.value
## 1         Training.condition  1        0.01    .922
## 2                    version  1   11.18 ***   <.001
## 3                agree_value  1 3772.95 ***   <.001
## 4 Training.condition:version  1        0.90    .344
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '+' 0.1 ' ' 1

Model outcome: table

Outcome: perceived credibility
Term Coefficient Std. Error Lower CI Upper CI p-value
(Intercept) 0.958 0.116 0.643 1.273 0.001
Training.condition1 -0.016 0.038 -0.091 0.058 0.665
version1 0.049 0.029 -0.008 0.106 0.091
Agreement with campaign 0.645 0.009 0.628 0.662 0.000
Training.condition1:version1 0.039 0.041 -0.042 0.119 0.344
Random Effect (id) 0.194
Random Effect (advert) 0.045
ICC 0.299
σ-squared 0.747
Marginal R-squared 0.546
Conditional R-squared 0.682

False discovery rate

Original and Adjusted p-Values for Fixed Effects
Fixed Effect Original P-Value Adjusted P-Value
(Intercept) 0.001 0.002
Training.condition1 0.665 0.665
version1 0.091 0.151
agree_value 0.000 0.000
Training.condition1:version1 0.344 0.430

Model without interaction

Below shows the specifics of the model with improved fit. AIC values and LRT indicated adding in an interaction term introduced overfitting. Without the interaction term, version is significant.

Outcome: perceived credibility
Term Coefficient Std. Error Lower CI Upper CI p-value
(Intercept) 0.948 0.115 0.633 1.264 0.001
Training.condition1 0.003 0.032 -0.059 0.065 0.923
version1 0.069 0.021 0.028 0.109 0.001
Agreement with campaign 0.645 0.009 0.628 0.662 0.000
Random Effect (id) 0.194
Random Effect (advert) 0.045
ICC 0.299
σ-squared 0.747
Marginal R-squared 0.546
Conditional R-squared 0.682
Original and Adjusted p-Values for Fixed Effects
Fixed Effect Original P-Value Adjusted P-Value
(Intercept) 0.001 0.001
Training.condition1 0.923 0.923
version1 0.001 0.001
agree_value 0.000 0.000

Plotting: model predictions

## Scale for y is already present.
## Adding another scale for y, which will replace the existing scale.

Assumptions

Each model is checked for the following assumptions for the model without an interaction effect:

  • normality of residuals
  • equal variance of residuals
  • normal distribution of residuals for the random effects

Credible model:

Shapiro-Wilk Test Results for Normality
Variable W_Statistic P_Value
id_effects 0.995 0.000
advert_effects 0.936 0.629
Note:
A p-value > 0.05 indicates the data is likely to be normally distributed. A p-value ≤ 0.05 suggests the data deviates from normality.

Exploratory: mediation model

This section relates to the following section of analysis reported in the paper:

As the dimensionality of the persuasion knowledge measures indicated the models loaded onto two separate factors, both were tested as a mediator for the association between version and credibility. The training condition is also included, due to its association with the persuasion knowledge variable.

This supports that credibility is indirectly influenced by the presence of a digital imprint through persuasion knowledge, through ‘formalising’ the feel of a post into an advertisement.

Fit Measures for CFA Model with all 4 measures
Measure Value
chisq 255.9568300
df 2.0000000
pvalue 0.0000000
rmsea 0.1549599
cfi 0.9835789
tli 0.9507366
Standardised Factor Loadings
Latent Variable Indicator Standardised Loading
credible_factor trustworthy_value 0.8689097
credible_factor accurate_value 0.9219665
credible_factor believable_value 0.8959943
credible_factor factual_value 0.6870868
Fit Measures for CFA Model with trustworthy, accuracy and believable measures
Measure Value
chisq 0
df 0
pvalue NA
rmsea 0
cfi 1
tli 1
Standardised Factor Loadings
Latent Variable Indicator Standardised Loading
credible_factor trustworthy_value 0.8564023
credible_factor accurate_value 0.9233735
credible_factor believable_value 0.9053528 
## lavaan 0.6-19 ended normally after 65 iterations
## 
##   Estimator                                         ML
##   Optimization method                           NLMINB
##   Number of model parameters                        22
## 
##   Number of observations                          5288
##   Number of clusters [id]                         1322
## 
## Model Test User Model:
##                                               Standard      Scaled
##   Test Statistic                               118.010      91.161
##   Degrees of freedom                                13          13
##   P-value (Chi-square)                           0.000       0.000
##   Scaling correction factor                                  1.295
##     Yuan-Bentler correction (Mplus variant)                       
## 
## Model Test Baseline Model:
## 
##   Test statistic                             25090.404   14852.697
##   Degrees of freedom                                25          25
##   P-value                                        0.000       0.000
##   Scaling correction factor                                  1.689
## 
## User Model versus Baseline Model:
## 
##   Comparative Fit Index (CFI)                    0.996       0.995
##   Tucker-Lewis Index (TLI)                       0.992       0.990
##                                                                   
##   Robust Comparative Fit Index (CFI)                         0.996
##   Robust Tucker-Lewis Index (TLI)                            0.992
## 
## Loglikelihood and Information Criteria:
## 
##   Loglikelihood user model (H0)             -37987.202  -37987.202
##   Scaling correction factor                                  1.850
##       for the MLR correction                                      
##   Loglikelihood unrestricted model (H1)     -37928.197  -37928.197
##   Scaling correction factor                                  1.643
##       for the MLR correction                                      
##                                                                   
##   Akaike (AIC)                               76018.403   76018.403
##   Bayesian (BIC)                             76163.013   76163.013
##   Sample-size adjusted Bayesian (SABIC)      76093.105   76093.105
## 
## Root Mean Square Error of Approximation:
## 
##   RMSEA                                          0.039       0.034
##   90 Percent confidence interval - lower         0.033       0.028
##   90 Percent confidence interval - upper         0.046       0.040
##   P-value H_0: RMSEA <= 0.050                    0.997       1.000
##   P-value H_0: RMSEA >= 0.080                    0.000       0.000
##                                                                   
##   Robust RMSEA                                               0.038
##   90 Percent confidence interval - lower                     0.031
##   90 Percent confidence interval - upper                     0.046
##   P-value H_0: Robust RMSEA <= 0.050                         0.995
##   P-value H_0: Robust RMSEA >= 0.080                         0.000
## 
## Standardized Root Mean Square Residual:
## 
##   SRMR                                           0.011       0.011
## 
## Parameter Estimates:
## 
##   Standard errors                        Robust.cluster
##   Information                                  Observed
##   Observed information based on                 Hessian
## 
## Latent Variables:
##                    Estimate  Std.Err  z-value  P(>|z|)
##   credibility =~                                      
##     trustworthy_vl    1.000                           
##     accurate_value    1.073    0.013   82.627    0.000
##     believable_val    1.103    0.014   81.114    0.000
##   latent_PK =~                                        
##     PK1_value         1.000                           
##     PK3_value         0.864    0.038   23.017    0.000
## 
## Regressions:
##                    Estimate  Std.Err  z-value  P(>|z|)
##   credibility ~                                       
##     version   (c1)    0.076    0.019    4.067    0.000
##     Trnng.cnd (c2)    0.014    0.027    0.529    0.597
##     agree_val (c3)    0.680    0.011   62.848    0.000
##   latent_PK ~                                         
##     version   (a1)    0.163    0.040    4.118    0.000
##     Trnng.cnd (a2)    0.213    0.071    2.981    0.003
##     agree_val (a3)   -0.228    0.021  -10.697    0.000
##   credibility ~                                       
##     latent_PK  (b)   -0.043    0.008   -5.398    0.000
## 
## Intercepts:
##                    Estimate  Std.Err  z-value  P(>|z|)
##    .trustworthy_vl    0.510    0.052    9.762    0.000
##    .accurate_value    0.814    0.051   15.810    0.000
##    .believable_val    1.003    0.058   17.329    0.000
##    .PK1_value         4.494    0.116   38.789    0.000
##    .PK3_value         4.208    0.099   42.304    0.000
## 
## Variances:
##                    Estimate  Std.Err  z-value  P(>|z|)
##    .trustworthy_vl    0.610    0.021   29.276    0.000
##    .accurate_value    0.280    0.018   15.619    0.000
##    .believable_val    0.455    0.026   17.405    0.000
##    .PK1_value         0.099    0.147    0.676    0.499
##    .PK3_value         0.790    0.115    6.896    0.000
##    .credibility       0.502    0.022   22.711    0.000
##    .latent_PK         3.308    0.156   21.242    0.000
## 
## Defined Parameters:
##                    Estimate  Std.Err  z-value  P(>|z|)
##     indirect_versn   -0.007    0.002   -3.279    0.001
##     indirect_trnng   -0.009    0.004   -2.536    0.011
##     indirect_grmnt    0.010    0.002    4.755    0.000
##     total_version     0.069    0.019    3.691    0.000
##     total_training    0.005    0.027    0.193    0.847
##     total_agreemnt    0.690    0.011   64.292    0.000

Hypothesis 5

Outcome: confidence in regulation

Does being informed explicitly about the purpose of digital imprints and their relation to regulatory compliance increase perceptions that political advertising is sufficiently regulated in the UK?

This model uses the training_df dataframe.

  • Outcome: confidence in regulation, numerical 1-7
  • Predictor: training condition

Assumptions of normality and equal variance of residuals are violated, due to the skewed distribution of the outcome variable. A robust standard errors model is fitted as a robustness check, and the same result is found, increasing confidence in the reliability of this estimate.

Plot: raw data

Model

Table

The top table shows the outcome using a traditional linear regression. The bottom table shows a robust regression model, weights generated using an M estimator with the ‘rlm’ function in the MASS package (loaded at this point in the script). The latter robustness check is included due to the violation of assumptions seen in the original model.

  Perceived sufficiency of advertising regulation
Predictors Estimates std. Error CI p
Intercept 3.01 0.06 2.90 – 3.12 <0.001
Trained -0.11 0.08 -0.27 – 0.04 0.150
Observations 1322
R2 / R2 adjusted 0.002 / 0.001
  Perceived sufficiency of advertising regulation (robust)
Predictors Estimates std. Error CI p
Intercept 2.88 0.06 2.76 – 3.01 <0.001
Trained -0.11 0.09 -0.29 – 0.07 0.215
Observations 1322

Plot: model predictions

Model assumptions

Pre-registered political goal hypotheses

Outcome: political goal

Did the presence of a digital imprint with a piece of campaign material increase participant awareness that the material had a political goal?

Plot: raw data

Below shows the skewed nature of the outcome variable, as well as the association separately for each advert.

## `geom_smooth()` using formula = 'y ~ x'

Pre-registration deviations

Pre-registered model:

  • model 1 <- lmer(political goal ~ imprint viewed + (1|id) + (1|advert))

Reported model:

  • model 1 <- lmer(political goal ~ version + training + (1|participant id) + (1|material))

Main deviation:

  • The effect of training was included as, although not central to the hypotheses, it was recognised the training may have also influenced perceptions of the outcome that needed to be accounted for to ensure the effect of version was not overestimated

Model fit metrics: AIC

Why was the reported model chosen out of theoretically feasible variants?

It is recognised a variety of random effect structures would be theoretically feasible. To choose the most appropriate model out of the alternatives, model fit criterion are used to assess if adding model complexity sacrifices valid estimation through reducing statistical power (Matuschek et al. 2017).

Theoretically feasible model structures:

  • Random intercepts: Outcome ~ 1 + version + training + (1|participant id) + (1|material)
  • Maximal random slopes model: Outcome ~ 1 + version + training + (1|participant id) + (1 + version|material)
  • Material:version intercepts model: Outcome ~ 1 + version + training + (1|participant id) + (1|material:version)
  • Fixed-effect interaction: Outcome ~ 1 + version*training + (1|participant id) + (1|material)

Although structure 2 achieved the lowest AIC, structure 1 was including random intercepts for participant ID and advert was chosen. This is because the model failed to converge as indicated by the ‘fit is singular’ warning message which was generated in response to structure 2. This indicated the model was unable to estimate the effect of the random slopes, likely due to the small size of the effect overall. As can be seen below in the model variation comparisons, the more complex structures 3 and 4 worsened the fit of the model when evaluated using AIC values. Model structure 1 was therefore deemed the best fit for the data, balancing model complexity with statistical power.

## boundary (singular) fit: see help('isSingular')
AIC Values and Variables: Political Goal
Variations Variables AIC
Structure 1 version + training + id (random intercept) + advert (random intercept) 14977.57
Structure 2 version + training + id (random intercept) + (version & advert (random slope)) 14974.59
Structure 3 version + training + id (random intercept) + (advert:version (random intercepts vary)) 14986.75
Structure 4 version*training + id (random intercept) + advert (random intercept) 14982.11

Model Outcomes: Table

Outcome: political goal
Term Coefficient Std. Error Lower CI Upper CI p-value
(Intercept) 6.014 0.251 5.226 6.802 0.000
version 0.045 0.024 -0.002 0.093 0.063
Training.condition -0.011 0.039 -0.087 0.065 0.780
Random Effect (id) 0.298
Random Effect (advert) 0.248
ICC 0.411
σ² 0.885
Marginal R² 0.000
Conditional R² 0.411

Plot: model predictions

Model assumptions

The assumptions checked below are as follows:

  • normality of residuals
  • variance of residuals
  • normality of residuals for each of the random effects (id and advert)

As can be seen, as the political goal variable is so heavily skewed, the assumptions for this model are not met. This is likely because the political nature of the posts could easy be inferred from the context of the post, not requiring a digital imprint to alter participants to this. Even though the validity of the predictions are called into question, it can be likely still be concluded that the political nature of the post was easy for participants to infer, making it unlikely a digital imprint would alter this perception in this context.

Shapiro-Wilk Test Results for Normality
Variable W_Statistic P_Value
id_effects 0.939 0.000
advert_effects 0.892 0.391
Note:
A p-value > 0.05 indicates the data is likely to be normally distributed. A p-value ≤ 0.05 suggests the data deviates from normality.

CLMM

## ---------------------------------------------------- 
## Test for     X2  df  probability 
## ---------------------------------------------------- 
## Omnibus          15.34   10  0.12
## version          9.77    5   0.08
## Training.condition   5.56    5   0.35
## ---------------------------------------------------- 
## 
## H0: Parallel Regression Assumption holds

Table 1: Fixed Effects for Ordinal Mixed-Effects Model
Odds_Ratio Estimate Std_Error Z_Value P_Value
1|2 0.001 -7.048 0.539 -13.067 < 0.001
2|3 0.005 -5.313 0.497 -10.686 < 0.001
3|4 0.012 -4.406 0.491 -8.977 < 0.001
4|5 0.034 -3.372 0.487 -6.923 < 0.001
5|6 0.157 -1.849 0.484 -3.816 < 0.001
6|7 1.835 0.607 0.483 1.256 0.209
version 1.095 0.091 0.056 1.611 0.107
Training.condition 0.973 -0.028 0.099 -0.278 0.781 
##                          2.5 %     97.5 %
## 1|2                -8.10457221 -5.9904790
## 2|3                -6.28754244 -4.3385136
## 3|4                -5.36764404 -3.4438972
## 4|5                -4.32698277 -2.4175030
## 5|6                -2.79845596 -0.8993882
## 6|7                -0.34037807  1.5544932
## version            -0.01963297  0.2006684
## Training.condition -0.22239784  0.1671614

Pre-registered Recall Hypotheses

Outcome: Recall of Names

Did viewing a digital imprint with a piece of campaign material increase participants memory of the campaigner name?

To test this, recall of the campaigner name can be tested with a logistical regression model to see if viewing the imprint boosted recall of the name.

Correct names:

  • Advert 1: Common sense collective: CSC
  • Advert 2: Breaking barriers alliance: BBA
  • Advert 3: Speak freely inc: SFI
  • Advert 4: Campaign for a better Britain: CBB

Plot: raw data

Pre-registration deviations

Pre-registered models:

  • Model 3 <- glm(campaigner name recall ~ imprint viewed, family = binomial())
  • Model 5 <- lm(correct campaigner name recall ~ training condition)

Reported model:

  • Model 3 <- glmer(recall ~ version*Training.condition + (1|id) + (1|advert))

Reason for deviation:

  • The pre-registered models were deemed inappropriate as, due to an oversight of the researcher when pre-registering, model 3 did not include random effects for participant and advertisement type. This was because the original models tested each advertisement separately. When the data was then merged for a combined analysis, although the random effects were planned, their inclusion was missed off the pre-registration document during the update. Additionally, for efficient reporting, there appeared no reason not to combine the models so both hypotheses could be tested at the same time. Thus, the following models were compared for their viability that tested both hypotheses 2a (pre-registered as 1b) and 2b (pre-registered as 2a), incorporated random effects structures of various model complexities and assessing model fit.

Model fit metrics: AIC

  • Random intercepts: Outcome ~ 1 + version + training + (1|participant id) + (1|material)
  • Maximal random slopes model: Outcome ~ 1 + version + training + (1|participant id) + (1 + version|material)
  • Material:version intercepts model: Outcome ~ 1 + version + training + (1|participant id) + (1|material:version)
  • Fixed-effect interaction: Outcome ~ 1 + version*training + (1|participant id) + (1|material)

As can be seen below, structure 4 achieved the lowest AIC value and was successfully able to converge, suggesting it provided the best model fit for the data. Supporting this further was a Log Likelihood Ratio test demonstrating that adding an interaction term to the model significantly improved model fit.

## boundary (singular) fit: see help('isSingular')
AIC Values and Variables: Recall of campaigner name
Variations Variables AIC
Structure 1 version + training + agreement + id (random intercept) + advert (random intercept) 6835.617
Structure 2 version + training + agreement + id (random intercept) + (version & advert (random slope)) 6838.047
Structure 3 version + training + agreement + id (random intercept) + (advert:version (random intercepts vary)) 6848.863
Structure 4 version*training + agreement + id (random intercept) + advert (random intercept) 6833.108 
## Contrasts set to contr.sum for the following variables: recall, advert
## Numerical variables NOT centered on 0: version, Training.condition
## If in interactions, interpretation of lower order (e.g., main) effects difficult.
## Mixed Model Anova Table (Type 3 tests, LRT-method)
## 
## Model: recall ~ version * Training.condition + (1 | id) + (1 | advert)
## Data: recall_df
## Df full model: 6
##                       Effect df  Chisq p.value
## 1                    version  1 4.61 *    .032
## 2         Training.condition  1   0.81    .369
## 3 version:Training.condition  1 4.51 *    .034
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '+' 0.1 ' ' 1

Model outcome: table

Below shows the statistically significant effect of the version of the advert viewed (i.e. if a digital imprint was present), as well as the interaction effect between version and training which initially indicated the training impacted how the campaigner name was evaluated by increasing attention towards the digital imprint.

Outcome: campaigner name recall
Term Coefficient Odds Ratio Lower CI (OR) Upper CI (OR) p-value
(Intercept) 0.022 1.022 0.553 1.891 0.944
version 0.183 1.201 1.017 1.418 0.031
Training.condition 0.083 1.087 0.907 1.302 0.366
version:Training.condition 0.257 1.293 1.021 1.637 0.033
Random Effect (id) 0.446
Random Effect (advert) 0.377
ICC 0.200
σ² 1.000
Marginal R² 0.010
Conditional R² 0.208

False Discovery Rate

However, when adjustments for multiple comparisons were made these effects lost statistical significance at the 0.05 threshold.

Original and Adjusted p-Values for Fixed Effects
Fixed Effect Original P-Value Adjusted P-Value
(Intercept) 0.944 0.944
version 0.031 0.066
Training.condition 0.366 0.489
version:Training.condition 0.033 0.066

Plot: model predictions

## Scale for y is already present.
## Adding another scale for y, which will replace the existing scale.
## Scale for colour is already present.
## Adding another scale for colour, which will replace the existing scale.

Model assumptions

The graphs below check for normality in the residuals of the model. A straight line suggests normality. As can be seen, in the random effects model (right) there is a heavier tail towards the middle, suggesting some variability in the data is not captured by the model. These are not strict assumptions that need to be met in a model that uses a binomial distribution, but are included to provide a full picture of the models fit.

Simple residual plot

Supplementary: Advert Variations

This part of the analysis relates to the following section of the paper:

“Overall, these robustness checks cast doubt on the consistency of these effects across different participants and material contents and formats. To provide more context, an exploratory analysis (reported in full in the supplementary materials) tested the effect of the digital imprint on each of the four experimental materials separately. It is noted that the total recall rates across the materials differed; material one was recalled correctly by 63.1% of participants, material two by 46.2%, material three by 37.7% and material four by 71.4%. The analysis indeed supported differences between the effect of digital imprints across different materials. For material two and three, those with a relatively lower recall rate overall, the presence of a digital imprint with the material resulted in significantly higher recall (material two: OR = 1.66, 95% CI [1.11, 2.47], p = <.001; material three: OR = 1.38, 95% CI [0.98, 1.94], p = 0.01). In contrast, for advert one and four, those with a relatively high recall rate overall, the presence of a digital imprint had no impact (material one: OR = 1.18, 95% CI [0.88, 1.57], p = 0.19; material four: OR = 1.28 , 95% CI [0.91, 1.78], p = 0.07).”

To test this, each correct campaign name was tested one by one with a logistical regression model to see if viewing the imprint boosted recall of the name. This results in 4 models and corresponding visualisations. This analysis uses the independent measures data frame with only one row per participant: training_df.

Correct names:

  • Advert 1: Common sense collective: CSC
  • Advert 2: Breaking barriers alliance: BBA
  • Advert 3: Speak freely inc: SFI
  • Advert 4: Campaign for a better Britain: CBB

Logs odds from the default model are converted to an odds ratio for easier interpretation. These are then presented as a table with the output of the regression. To understand the direction of the odds ratio, check the original log odds coefficient.

  CSC
Predictors Odds Ratios CI p
(Intercept) 1.58 1.35 – 1.85 <0.001
Advert 1 [1] 1.18 0.94 – 1.47 0.154
Observations 1322
R2 Tjur 0.002
AIC 1743.022
Recall of Common Sense Collective campaign group by imprint viewed
Coefficient SE Odds ratio 95% CI(lower) 95% CI(upper)
(Intercept) 0.46 0.08 1.58 1.35 1.85
Imprint viewed with material 0.16 0.11 1.18 0.94 1.47
  BBA
Predictors Odds Ratios CI p
(Intercept) 0.69 0.59 – 0.80 <0.001
Advert 2 [1] 1.55 1.24 – 1.92 <0.001
Observations 1322
R2 Tjur 0.012
AIC 1813.606
Recall of Breaking Barriers Alliance campaign group by imprint viewed
Coefficient SE Odds ratio 95% CI(lower) 95% CI(upper)
(Intercept) -0.37 0.08 0.69 0.59 0.80
Imprint viewed with material 0.44 0.11 1.55 1.24 1.92
  SFI
Predictors Odds Ratios CI p
(Intercept) 0.51 0.43 – 0.60 <0.001
Advert 3 [1] 1.39 1.11 – 1.74 0.004
Observations 1322
R2 Tjur 0.006
AIC 1747.146
Recall of Speak Freely Inc campaign group by imprint viewed
Coefficient SE Odds ratio 95% CI(lower) 95% CI(upper)
(Intercept) -0.67 0.08 0.51 0.43 0.60
Imprint viewed with material 0.33 0.11 1.39 1.11 1.74
  CBB
Predictors Odds Ratios CI p
(Intercept) 2.28 1.94 – 2.70 <0.001
Advert 4 [1] 1.20 0.95 – 1.52 0.135
Observations 1322
R2 Tjur 0.002
AIC 1584.110
Recall of common sense collective campaign group by imprint viewed
Coefficient SE Odds ratio 95% CI(lower) 95% CI(upper)
(Intercept) 0.83 0.08 2.28 1.94 2.70
Imprint viewed with material 0.18 0.12 1.20 0.95 1.52

Supplementary: plots

References